Tải bản đầy đủ

1491945672 {7782C3BF} learning MCollective parallel server management in puppet and chef rhett 2014 08 31



LearningMCollective
JoRhett



LearningMCollective
byJoRhett
Copyright©2014JoRhett.Allrightsreserved.
PrintedintheUnitedStatesofAmerica.
PublishedbyO’ReillyMedia,Inc.,1005GravensteinHighwayNorth,Sebastopol,CA
95472.
O’Reillybooksmaybepurchasedforeducational,business,orsalespromotionaluse.
Onlineeditionsarealsoavailableformosttitles(http://safaribooksonline.com).Formore
information,contactourcorporate/institutionalsalesdepartment:800-998-9938or
corporate@oreilly.com.
Editors:CourtneyNashandBrianAnderson
ProductionEditor:KaraEbrahim
Copyeditor:JasmineKwityn
Proofreader:AmandaKersey

Indexer:JudyMcConville
InteriorDesigner:DavidFutato
CoverDesigner:EllieVolckhausen
Illustrator:RebeccaDemarest
August2014:FirstEdition


RevisionHistoryfortheFirstEdition
2014-08-11:FirstRelease
Seehttp://oreilly.com/catalog/errata.csp?isbn=9781491945674forreleasedetails.
NutshellHandbook,theNutshellHandbooklogo,andtheO’Reillylogoareregistered
trademarksofO’ReillyMedia,Inc.LearningMCollective,theimageofEnglishLeicester
sheep,andrelatedtradedressaretrademarksofO’ReillyMedia,Inc.
Manyofthedesignationsusedbymanufacturersandsellerstodistinguishtheirproducts
areclaimedastrademarks.Wherethosedesignationsappearinthisbook,andO’Reilly
Media,Inc.,wasawareofatrademarkclaim,thedesignationshavebeenprintedincapsor
initialcaps.
Whileeveryprecautionhasbeentakeninthepreparationofthisbook,thepublisherand
authorassumenoresponsibilityforerrorsoromissions,orfordamagesresultingfromthe
useoftheinformationcontainedherein.
978-1-491-94567-4
[LSI]



Preface
ThisbookwillteachyoutoinstallandusetheMarionetteCollective,hereafterreferredto
asMCollective.ItwilloutlinehowMCollectiveworksandhowMCollective’sdesign
providesvaluetoyou.You’lllearnhowtoseamlesslyorchestratechangeonthousandsof
nodesworldwideoronahandfulofnodeswithaspecificcharacteristicjustaseasily.
Thisbookprovidesspecificinstructiononhowtouseconfigurationmanagementtools
likePuppetandCheftodeployMCollective.ItcovershowMCollectivecanmanipulate
thePuppetandChefagentsandusedataprovidedbythem.


WhoThisBookIsFor
ThisbookisprimarilyaimedatsystemadministratorsandoperationsorDevOps
engineers.Ifyouareresponsiblefordevelopmentorproductionnodes,thisbookwill
provideyouwithusefultoolstomakeyourjobeasierthaneverbefore.Ifyouareusing
PuppetorCheftomanageyournodes,you’regoingtolearnhowMCollectivesnapsinto

yourexistingconfigurationmanagementtogiveyouinstantcontrolofyourmanaged
nodes.Withinamonth,you’llwonderhowyouevergotalongwithoutit.
Nomatterwhatyoucallyourself,ifyoufeelthatyouspendtoomuchtimemanaging
computers,thenthisbookisforyou.You’dliketogetitdonefastersoyoucanfocuson
somethingelse.You’dliketodoitmoreconsistently,sothatyoudon’thavetochase
downone-offproblemsinyourreports.Oryou’vegotsomenewdemandsthatyou’re
lookingforawaytosolve.Ifanyofthesestatementsfit,youwillfindMCollectivetobe
oneofthebesttoolsinyourtoolbox.


WhattoExpectfromMe
Thisbookwillnotbeatomefilledwithreferencematerialirrelevanttotheday-to-day
systemadministrator—exactlytheopposite.Throughoutthisbook,wewillneverstray
fromonesimplegoal:wefocusalloureffortsonhowMCollectivecanhelpyoudo
somethingfasterorbetterthaneverbefore.
Thisbookwillnevertellyoutorunascriptandnottellyouwhatitdoes,orwhy.Ihate
modelingsystemstodeterminewhataninstallationscriptdid,andIwon’tdothistoyou.
Inthisbook,youwillbuilduptheentireinstallationbyhand.You’llknowwhereevery
configurationfilelives.You’lllearneveryconfigurationparameterandwhatitmeans.
Andyes,thenyouwilllearnthePuppetmodulesandChefcookbooksyoucanuseto
automatedeploymentseamlesslythroughoutyourenvironment.


WhatYouWillNeed
YoumayuseanymodernLinux,Unix,Mac,orWindowssystemandsuccessfullyfollow
thehands-ontutorialsinthisbook.
Althoughwe’llintroduceawebclientforMCollective,themajorityoftheprocessof
configuringandenablingMCollectiveandutilizationofclientappswillbeperformed
throughthecommandline.
Abeginnertosystemadministrationcanfolloweverytutorialinthisbook.Any
experiencewithscripts,coding,orconfigurationmanagementwillenhancewhatyoucan
getoutofthisbook,aswewillspendsometimedocumentinghowMCollectivecan
utilizeandenhanceeachofthose.
PartIIIdocumentshowtobuildcustompluginsforMCollectiveintheRubylanguage.
Rubyprogrammerswillbeabletoutilizethisimmediately,whileothersmayneed
referencematerials—suchasMichaelFitzgerald’sLearningRuby(O’Reilly)—asthey
addmorefeaturestotheworkingexamplesprovidedhere.


WhatYou’llFindinThisBook
Chapter1discusseswhatMCollectivedoes,howitworks,andhowitcanbeusedto
orchestratechangeonyoursystemsfasterandeasierthanyoucouldhaveimagined.Learn
howMCollectiveisdifferentfromcontrolsystemsthatloopthrougheachtargetandhow
trueparallelexecutioncanbenefityourenvironment.
TheremainderofPartIwillfocusongettingyouupandrunningwithaworking
MCollectiveinstallation.YouwilllearnthecomponentsthatmakeuptheMCollective
infrastructure.You’llinstallandconfigureeachinamannersuitableforyourspecific
environment.
Thiswon’tbeatestenvironmentfortrainingthatdoesn’tmatchyourrealconcerns;
instead,you’llperformrealoperationsonhoststhatmatchyourproductionenvironment.
You’llseehoweasyitistodeployMCollectiveandexactlyhowpowerfulthetoolsit
providesare.
PartIItakesyouonanuts-and-boltstourinsideMCollective’sarchitecture,backbone,
transport,andsecuritycontrols.You’lllearnaboutusinganetworkofbrokerstoresolve
multisiteorredundancyrequirements.You’lllearnhowtocreateandusecollectivesto
handlethousandsofMCollectiveagentsspreadaroundtheworld.Afterfinishingthis
section,you’llbeabletofine-tuneMCollectiveforyourexactenvironment:smallbut
globallydiverse,immenseinscalebutlocalized,oracombinationofboth.
MCollectivehasanactivedeveloperandusercommunity.“FindingCommunityPlugins”
directsyoutoonlinerepositoriesofclientsandagentsbuiltbyothers,aswellasconcrete
examplesofhowtouseothers’pluginsinyourenvironment.
InPartIII,youwillcreateyourownserverandclientpluginstoperformanyactionyou
canconceiveof.You’lllearnhowtocreateapplicationclientsandhowtocreatelisteners
tocollectregistrationdetailsfromtheagentsystems.Bestofall,thesecretsofcollecting
andprocessingresponsesusingadirectedreplywillallowyoutocreateself-healing
systems.


HowtoUseThisBook
ThisbookprovidesexplicitinstructionsforconfiguringandusingMCollectivefromthe
commandlinewithouttheuseofanexternaltools.
ThebookdocumentsandutilizesaPuppetmodulethatcanimplementandcontrolevery
featureofMCollectivedocumentedinthisbook.InPartII,everyconfigurationoptionis
documentedforbothstandaloneandPuppetconfiguration.
ThebookdocumentsaChefcookbookthatcanbeusedtomaintainMCollectiveandgives
MCollectivetheabilitytomanagetheChefagent.
IfyouuseSalt,Cfengine,oranyotherconfiguration-managementsystem,theinstructions
herecanbeusedtodeployMCollective.Youwillfinditeasytocreateconfiguration
policiesfromtheexamplesinthisbook.TheserverpluginprovidedinPartIII,alongwith
thesectionabouthowtointeractwithexternalcommands,couldbeeasilyadjustedto
controlthemanagementagentoneachnode.


IPv6Ready
EveryexamplewithIPaddresseswillincludebothIPv4andIPv6statements.Ifyou’re
onlyusingoneoftheseprotocols,youcanignoretheother.MCollectivewillhappilyuse
anycombinationofthem.MoredetailsaboutcomplexIPv6setupswillbecoveredin
“IPv6Dual-StackEnvironments”.


ConventionsUsedinThisBook
Thefollowingtypographicalconventionsareusedinthisbook:
Italic
Indicatesnewterms,URLs,emailaddresses,filenames,andfileextensions.
Constantwidth

Usedforprogramlistings,aswellaswithinparagraphstorefertoprogramelements
suchasvariableorfunctionnames,databases,datatypes,environmentvariables,
statements,andkeywords.
Constantwidthbold

Showscommandsorothertextthatshouldbetypedliterallybytheuser.
Constantwidthitalic

Showstextthatshouldbereplacedwithuser-suppliedvaluesorbyvaluesdetermined
bycontext.
Tip
Thiselementsignifiesatiporsuggestion.
Note
Thiselementsignifiesageneralnote.
Warning
Thiselementindicatesawarningorcaution.


UsingCodeExamples
Supplementalmaterial(codeexamples,exercises,etc.)isavailablefordownloadat
https://github.com/jorhett/learning-mcollective.
Thisbookisheretohelpyougetyourjobdone.Ingeneral,ifexamplecodeisoffered
withthisbook,youmayuseitinyourprogramsanddocumentation.Youdonotneedto
contactusforpermissionunlessyou’rereproducingasignificantportionofthecode.For
example,writingaprogramthatusesseveralchunksofcodefromthisbookdoesnot
requirepermission.SellingordistributingaCD-ROMofexamplesfromO’Reillybooks
doesrequirepermission.Answeringaquestionbycitingthisbookandquotingexample
codedoesnotrequirepermission.Incorporatingasignificantamountofexamplecode
fromthisbookintoyourproduct’sdocumentationdoesrequirepermission.
Weappreciate,butdonotrequire,attribution.Anattributionusuallyincludesthetitle,
author,publisher,andISBN.Forexample:“LearningMCollectivebyJoRhett(O’Reilly).
Copyright2014JoRhett,978-1-491-94567-4.”
Ifyoufeelyouruseofcodeexamplesfallsoutsidefairuseorthepermissiongivenabove,
feelfreetocontactusatpermissions@oreilly.com.


Safari®BooksOnline
SafariBooksOnlineisanon-demanddigitallibrarythatdeliversexpertcontentinboth
bookandvideoformfromtheworld’sleadingauthorsintechnologyandbusiness.
Technologyprofessionals,softwaredevelopers,webdesigners,andbusinessandcreative
professionalsuseSafariBooksOnlineastheirprimaryresourceforresearch,problem
solving,learning,andcertificationtraining.
SafariBooksOnlineoffersarangeofplansandpricingforenterprise,government,
education,andindividuals.
Membershaveaccesstothousandsofbooks,trainingvideos,andprepublication
manuscriptsinonefullysearchabledatabasefrompublisherslikeO’ReillyMedia,
PrenticeHallProfessional,Addison-WesleyProfessional,MicrosoftPress,Sams,Que,
PeachpitPress,FocalPress,CiscoPress,JohnWiley&Sons,Syngress,Morgan
Kaufmann,IBMRedbooks,Packt,AdobePress,FTPress,Apress,Manning,NewRiders,
McGraw-Hill,Jones&Bartlett,CourseTechnology,andhundredsmore.Formore
informationaboutSafariBooksOnline,pleasevisitusonline.


HowtoContactUs
Pleaseaddresscommentsandquestionsconcerningthisbooktothepublisher:
O’ReillyMedia,Inc.
1005GravensteinHighwayNorth
Sebastopol,CA95472
800-998-9938(intheUnitedStatesorCanada)
707-829-0515(internationalorlocal)
707-829-0104(fax)
Wehaveawebpageforthisbook,wherewelisterrata,examples,andanyadditional
information.Youcanaccessthispageathttp://bit.ly/learn-mcollective.
Tocommentorasktechnicalquestionsaboutthisbook,sendemailto
bookquestions@oreilly.com.
Formoreinformationaboutourbooks,courses,conferences,andnews,seeourwebsiteat
http://www.oreilly.com.
FindusonFacebook:http://facebook.com/oreilly
FollowusonTwitter:http://twitter.com/oreillymedia
WatchusonYouTube:http://www.youtube.com/oreillymedia


Acknowledgments
IowesignificantgratitudetoR.I.Pienaar,whocreatedMCollectiveandcontinuesto
providevaluableassistanceonsupportchannels.Thisbookwouldneverhavebeen
possiblewithouthisdirectandindirectassistance.
I’dliketothankRichardClampandPeterLoubser,whoprovidethevisiblesupportand
ongoingdevelopmentfromPuppetLabs.
TheChefportionsofthisbookwouldn’thavebeenpossiblewithouttheongoing
developmentoftheMCollectiveCookbookbyZacStevens.HeandMischaTaylorof
Chefbothprovidedinvaluableassistanceintheirpersonaltime.
Ioweadrinkandmuchthankstothemanypeoplewhoprovidedinputandfeedbackon
thebookduringthewritingprocess,includingbutdefinitelynotlimitedtothetechnical
reviewers,RyanDill(StubHub)andJenniferDavis(Chef).
Andfinally,I’dliketothankmyO’Reillyeditors,CourtneyNashandBrianAnderson,
whogavemeexcellentguidanceonthebookandwereapleasuretoworkwiththroughout
theproject.


PartI.GettingStarted
WewillstartthispartwithanoverviewofwhatMCollectivedoes,howitworks,andhow
itcanbeusedtoorchestratechange.We’lldiscusshowMCollectivediffersfromcontrol
systemsthatloopthrougheachtarget,andhowtrueparallelexecutioncanbenefityour
environment.
Soundsabitboring,huh?Takeamomentandenjoyit,becausefromthatpointonward,
you’regoingtobeoperatinglive.It’sallhands-onfromhere.
You’llperformarealinstallationofMCollectiveserversandclientsinyourenvironment.
Nodemosystem,notinyconfigurationthatdoesn’tmatchtoyourneeds.You’llbuilda
workingMCollectiveinstallationandtestitoutforyourexactneeds.You’llusetheclient
programtomakelivebutnonoperationalcallsthatarespecificanduniquetoyourown
servers.
I’llcovernetworkandinfrastructurerequirementsforMCollectiveandhowtoconfirm
thateachisconfiguredproperly.You’llgetin-depthinstructiononcommoninstallation
problemsandlearntofixtheseandrelatedissuesonyourown.
Youcanuseconfiguration-managementtoolstoinstallandconfigureMCollective.We’ll
introduceacompanionPuppetmodulethatiscapableofdeployinggloballywithminimal
configuration.IfyouusePuppetorChef,you’llinstallanMCollectiveagenttocontrolit.
PuppetandChefagentswillstopbeingsomethingthatrunsperiodicallyandinstead
becomeinteractiveresourcesyoucanutilizeforimmediatechange.AllthisinjustPartI
ofthebook!



Chapter1.Introduction


WhatIsMCollective?
MCollectiveprovidesaframeworkforparalleljobexecution.Itiscommonlyusedto
orchestratechangeacrossclustersofserversinnearrealtime.Itisnotentirelyinaccurate
toimaginetheclassicmarionettecontrollerwithpuppetsdancingonstrings.(Yesit’sa
pun,butitismoreaptthanyoumayrealize.)
MCollectiveisanadjuncttoolinyourtoolboxthatcooperatesandenhancesthe
capabilitiesofconfigurationmanagementtoolslikePuppet,Chef,andSalt.Whereasthese
toolsanalyzeandacttoensurecompleteconfigurationconsistency,MCollective
orchestratesspecificandoftensingularactionsacrosssystemssignificantlyfaster.
Let’stalkaboutthedifferencebetweenMCollectiveandusingChef,Puppet,Capistrano,
SaltOverstack,orhand-builttoolsfororchestration.Youmayhaveusedaparallel
executioncommandlikepsshbefore.Youmayhavebuiltorimprovedoneyourself.And
ifyouhave,you’refamiliarwiththeirlimitations,whichincludeanyorallofthe
following:
Loopsthroughsystemsinorder,processingafewatatime
Simplistic(oroverlycomplex)authenticationmechanisms
Requirescustomizationforeachalternativeenvironment
Unabletorespondtodeviationsinresponse
Tooeasytooverlookfatalerrormessagesoutputtothescreen
Doesn’tsupportorextendexistingmanagementtools
MCollectivediffersfromthesetoolsinawidevarietyofways:
Usesamasterlessenvironmentthatallowsparallelexecutiononthousandsofsystems
Allowsforcustomauthenticationandauthorizationmechanisms
Handlesdifferentplatforms,architectures,andlocalenvironmentstransparently
Returnsfulldatasetsasresultcodes,allowingintelligentresponse
Directsresultstoaprocessorthattakesactiononresponses
IntegratescleanlywithconfigurationmanagementtoolslikePuppetandChef
Let’stalkabouthowMCollective’sdesignmakesthispossible.


WhyParallelExecution?
Allofushavespenttimetryingtomakesomethinghappenatthesametimeonanumber
ofsystems.Howmanytimeshaveyoudonesomethinglikethis?
$forhostinbunchofhosts
do
scpconfig-file$host:/some/path
ssh$host"serviceapacherestart"
done

Youtellyourselfthatthisishappeningallatonce,butyouknowbetter.Evenwithmore
complexparallelexecutionprocessorslikepssh,ClusterSSH,orFabric,thesequential
orderingensuresadelaythatcreatesdriftbetweenhosts.
Ifyouhavemorethanafewhosts,theoutputfromthecommandsscrollsoffthescreen.
Didyounoticethatthe15thhostcommandfailed?ThebetterparallelSSHprocessorswill
keeptheoutputfromeachhost,butyouneedtoexamineitforerrors.Noneofthemcan
identifyanerrorinthemiddleofamulticommandsequenceandbringittoyourattention.
Whenit’stimetodosomethingonalotofhosts,youwantittohappenfast,andyouneed
toknowthatitsucceeded.
Note
Puppet(orChef,Salt,etc.)usersmightwanttostanduprightnowtosay,“ButIcando
this!Myconfigurationagentcanensurethatthesethingshappenonawholebunchof
systemsallatonce!”
You’reright,Puppet/Chef/etc.aregreattoolsformakingchangesonsystemsandensuring
thatthosechangeshappen.Buttheydon’tmakethishappenonmanysystemsatthesame
time.Eachagentpullsfromtheserverperiodically,allowingtheservertoprocessafewat
atime.Theagentsreceivetheirconfigurationchangesspreadoutovertimeandthen
combinemanydifferentinputstoproduceaconsistentconfiguration.Thisresolution
processusuallytakesatleast30secondsandoftenmanyminutes.
Inveryfewenvironmentscouldeveryconfigurationmanagementagentgetacatalogfrom
theserverandexecuteitatthesamemoment.Additionally,eachagentwouldtake
differingtimestoprocesstheircatalogs.MCollectiveistheperfectcomplementfor
configurationmanagementagents,designedspecificallytoorchestrateactionsquickly
acrossmanynodes.


HowMCollectiveWorks
MCollectivewasdesignedfromthegrounduptoachievetrueparallelexecutionwith
consistentandrepeatableresults.MCollectiveavoidstheuseofacentralizedmasterfor
commandandcontrol,thusavoidingcentralizedresourceproblems.Italsodoesn’treach
outtotheclientsinanorderedloop,thusavoidingdriftbetweeneachofthesystems.
MCollectiveusespublish/subscribemiddlewaretotransportrequestsbetweenclientsand
servers.Controllednodesrunanapplicationservernamedmcollectived*.Thisserver
subscribestomessagetopics.Clientsareapplicationsthatpublishrequeststothemessage
topics.Thepublishandsubscribeoperationsaredonethroughpersistentconnectionstoa
middlewarebroker.
Themcollectivedserverregisterswiththemiddlewarebrokerandremainsinalistening
orIDLEstate.Wheneveraclientsendsarequesttothemiddleware,eachserverreceives
andevaluatestherequestimmediatelyandindependently.mcollectivedvalidatesthe
requestandthenhandsitofftoanagenttoprocesstherequest.Theagentprocessesthe
requestandsendsthereplyback.Allresourcesconsumedarelocaltothenodewithout
anypullfromorpushtoacentralizedresource,likeaPuppetmasterorChefserver.
Inthismodel,youcanhaveacommandexecuteontens,hundreds,orthousandsofnodes
atexactlythesametime.Thispublish/subscribeinfrastructuredeliversascalableandfast
parallelexecutionenvironment.ThemodelisillustratedinFigure1-1.

Figure1-1.Theone-to-manypublishingmodelusedbyMCollective
Nowyoumightbethinkingtoyourself,“WhatifIonlywantthecommandexecutedona
subsetofnodes?”MCollectiveprovidesarichlanguagefordescribingwhichnodes
shouldexecutethecommands.Youcansendfiltersbasedonhostname,operatingsystem,
packagesinstalled,processesrunning,andmanyothercriteria.Bestofall,newcriteria
customtoyourenvironmentwillbeavailablewhenyoucreateyourownagent.


Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Tải bản đầy đủ ngay

×